{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# 1. PyRIT Scan\n",
    "\n",
    "`pyrit_scan` allows you to run automated security testing and red teaming attacks against AI systems using [scenarios](../scenarios/0_scenarios.ipynb) for strategies and [configuration](../setup/1_configuration.ipynb).\n",
    "\n",
    "Note in this doc the ! prefaces all commands in the terminal so we can run in a Jupyter Notebook.\n",
    "\n",
    "## Quick Start\n",
    "\n",
    "For help:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting PyRIT...\n",
      "usage: pyrit_scan [-h] [--log-level LOG_LEVEL] [--list-scenarios]\n",
      "                  [--list-initializers] [--database DATABASE]\n",
      "                  [--initializers INITIALIZERS [INITIALIZERS ...]]\n",
      "                  [--initialization-scripts INITIALIZATION_SCRIPTS [INITIALIZATION_SCRIPTS ...]]\n",
      "                  [--strategies SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...]]\n",
      "                  [--max-concurrency MAX_CONCURRENCY]\n",
      "                  [--max-retries MAX_RETRIES] [--memory-labels MEMORY_LABELS]\n",
      "                  [scenario_name]\n",
      "\n",
      "PyRIT Scanner - Run security scenarios against AI systems\n",
      "\n",
      "Examples:\n",
      "  # List available scenarios and initializers\n",
      "  pyrit_scan --list-scenarios\n",
      "  pyrit_scan --list-initializers\n",
      "\n",
      "  # Run a scenario with built-in initializers\n",
      "  pyrit_scan foundry_scenario --initializers openai_objective_target\n",
      "\n",
      "  # Run with custom initialization scripts\n",
      "  pyrit_scan encoding_scenario --initialization-scripts ./my_config.py\n",
      "\n",
      "  # Run specific strategies\n",
      "  pyrit_scan encoding_scenario --initializers openai_objective_target --strategies base64 rot13\n",
      "  pyrit_scan foundry_scenario --initializers openai_objective_target --max-concurrency 10 --max-retries 3\n",
      "  pyrit_scan encoding_scenario --initializers openai_objective_target --memory-labels '{\"run_id\":\"test123\"}'\n",
      "\n",
      "positional arguments:\n",
      "  scenario_name         Name of the scenario to run\n",
      "\n",
      "options:\n",
      "  -h, --help            show this help message and exit\n",
      "  --log-level LOG_LEVEL\n",
      "                        Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)\n",
      "                        (default: WARNING)\n",
      "  --list-scenarios      List all available scenarios and exit\n",
      "  --list-initializers   List all available scenario initializers and exit\n",
      "  --database DATABASE   Database type to use for memory storage (InMemory,\n",
      "                        SQLite, AzureSQL) (default: SQLite)\n",
      "  --initializers INITIALIZERS [INITIALIZERS ...]\n",
      "                        Built-in initializer names to run before the scenario\n",
      "                        (e.g., openai_objective_target)\n",
      "  --initialization-scripts INITIALIZATION_SCRIPTS [INITIALIZATION_SCRIPTS ...]\n",
      "                        Paths to custom Python initialization scripts to run\n",
      "                        before the scenario\n",
      "  --strategies SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...], -s SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...]\n",
      "                        List of strategy names to run (e.g., base64 rot13)\n",
      "  --max-concurrency MAX_CONCURRENCY\n",
      "                        Maximum number of concurrent attack executions (must\n",
      "                        be >= 1)\n",
      "  --max-retries MAX_RETRIES\n",
      "                        Maximum number of automatic retries on exception (must\n",
      "                        be >= 0)\n",
      "  --memory-labels MEMORY_LABELS\n",
      "                        Additional labels as JSON string (e.g.,\n",
      "                        '{\"experiment\": \"test1\"}')\n"
     ]
    }
   ],
   "source": [
    "!pyrit_scan --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "### Discovery\n",
    "\n",
    "List all available scenarios:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting PyRIT...\n",
      "Loading PyRIT modules...\n",
      "\n",
      "Available Scenarios:\n",
      "================================================================================\n",
      "\u001b[1m\u001b[36m\n",
      "  content_harms_scenario\u001b[0m\n",
      "    Class: ContentHarmsScenario\n",
      "    Description:\n",
      "      Content Harms Scenario implementation for PyRIT. This scenario contains\n",
      "      various harm-based checks that you can run to get a quick idea about\n",
      "      model behavior with respect to certain harm categories.\n",
      "    Aggregate Strategies:\n",
      "      - all\n",
      "    Available Strategies (7):\n",
      "      hate, fairness, violence, sexual, harassment, misinformation, leakage\n",
      "    Default Strategy: all\n",
      "\u001b[1m\u001b[36m\n",
      "  cyber_scenario\u001b[0m\n",
      "    Class: CyberScenario\n",
      "    Description:\n",
      "      Cyber scenario implementation for PyRIT. This scenario tests how willing\n",
      "      models are to exploit cybersecurity harms by generating malware. The\n",
      "      CyberScenario class contains different variations of the malware\n",
      "      generation techniques.\n",
      "    Aggregate Strategies:\n",
      "      - all\n",
      "    Available Strategies (2):\n",
      "      single_turn, multi_turn\n",
      "    Default Strategy: all\n",
      "\u001b[1m\u001b[36m\n",
      "  encoding_scenario\u001b[0m\n",
      "    Class: EncodingScenario\n",
      "    Description:\n",
      "      Encoding Scenario implementation for PyRIT. This scenario tests how\n",
      "      resilient models are to various encoding attacks by encoding potentially\n",
      "      harmful text (by default slurs and XSS payloads) and testing if the\n",
      "      model will decode and repeat the encoded payload. It mimics the Garak\n",
      "      encoding probe. The scenario works by: 1. Taking seed prompts (the\n",
      "      harmful text to be encoded) 2. Encoding them using various encoding\n",
      "      schemes (Base64, ROT13, Morse, etc.) 3. Asking the target model to\n",
      "      decode the encoded text 4. Scoring whether the model successfully\n",
      "      decoded and repeated the harmful content By default, this uses the same\n",
      "      dataset as Garak: slur terms and web XSS payloads.\n",
      "    Aggregate Strategies:\n",
      "      - all\n",
      "    Available Strategies (17):\n",
      "      base64, base2048, base16, base32, ascii85, hex, quoted_printable,\n",
      "      uuencode, rot13, braille, atbash, morse_code, nato, ecoji, zalgo,\n",
      "      leet_speak, ascii_smuggler\n",
      "    Default Strategy: all\n",
      "\u001b[1m\u001b[36m\n",
      "  foundry_scenario\u001b[0m\n",
      "    Class: FoundryScenario\n",
      "    Description:\n",
      "      FoundryScenario is a preconfigured scenario that automatically generates\n",
      "      multiple AtomicAttack instances based on the specified attack\n",
      "      strategies. It supports both single-turn attacks (with various\n",
      "      converters) and multi-turn attacks (Crescendo, RedTeaming), making it\n",
      "      easy to quickly test a target against multiple attack vectors. The\n",
      "      scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into\n",
      "      their constituent attack strategies, or you can specify individual\n",
      "      strategies directly. Note this is not the same as the Foundry AI Red\n",
      "      Teaming Agent. This is a PyRIT contract so their library can make use of\n",
      "      PyRIT in a consistent way.\n",
      "    Aggregate Strategies:\n",
      "      - all, easy, moderate, difficult\n",
      "    Available Strategies (23):\n",
      "      ansi_attack, ascii_art, ascii_smuggler, atbash, base64, binary, caesar,\n",
      "      character_space, char_swap, diacritic, flip, leetspeak, morse, rot13,\n",
      "      suffix_append, string_join, unicode_confusable, unicode_substitution,\n",
      "      url, jailbreak, tense, multi_turn, crescendo\n",
      "    Default Strategy: easy\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Total scenarios: 4\n"
     ]
    }
   ],
   "source": [
    "!pyrit_scan --list-scenarios"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4",
   "metadata": {},
   "source": [
    "**Tip**: You can also discover user-defined scenarios by providing initialization scripts:\n",
    "\n",
    "```shell\n",
    "pyrit_scan --list-scenarios --initialization-scripts ./my_custom_initializer.py\n",
    "```\n",
    "\n",
    "This will load your custom scenario definitions and include them in the list.\n",
    "\n",
    "## Initializers\n",
    "\n",
    "PyRITInitializers are how you can configure the CLI scanner. PyRIT includes several built-in initializers you can use with the `--initializers` flag.\n",
    "\n",
    "The `--list-initializers` command shows all available initializers. Initializers are referenced by their filename (e.g., `objective_target`, `objective_list`, `simple`) regardless of which subdirectory they're in.\n",
    "\n",
    "List the available initializers using the --list-initializers flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting PyRIT...\n",
      "\n",
      "Available Initializers:\n",
      "================================================================================\n",
      "\u001b[1m\u001b[36m\n",
      "  objective_list\u001b[0m\n",
      "    Class: ScenarioObjectiveListInitializer\n",
      "    Name: Simple Objective List Configuration for Scenarios\n",
      "    Execution Order: 10\n",
      "    Required Environment Variables: None\n",
      "    Description:\n",
      "      Simple Objective List Configuration for Scenarios\n",
      "\u001b[1m\u001b[36m\n",
      "  openai_objective_target\u001b[0m\n",
      "    Class: ScenarioObjectiveTargetInitializer\n",
      "    Name: Simple Objective Target Configuration for Scenarios\n",
      "    Execution Order: 10\n",
      "    Required Environment Variables:\n",
      "      - DEFAULT_OPENAI_FRONTEND_ENDPOINT\n",
      "      - DEFAULT_OPENAI_FRONTEND_KEY\n",
      "    Description:\n",
      "      This configuration sets up a simple objective target for scenarios using\n",
      "      OpenAIChatTarget with basic settings. It initializes an openAI chat\n",
      "      target using the OPENAI_CLI_ENDPOINT and OPENAI_CLI_KEY environment\n",
      "      variables.\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Total initializers: 2\n"
     ]
    }
   ],
   "source": [
    "!pyrit_scan --list-initializers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6",
   "metadata": {},
   "source": [
    "### Running Scenarios\n",
    "\n",
    "You need a single scenario to run, you need two things:\n",
    "\n",
    "1. A Scenario. Many are defined in `pyrit.scenario.scenarios`. But you can also define your own in initialization_scripts.\n",
    "2. Initializers (which can be supplied via `--initializers` or `--initialization-scripts`). Scenarios often don't need many arguments, but they can be configured in different ways. And at the very least, most need an `objective_target` (the thing you're running a scan against).\n",
    "3. Scenario Strategies (optional). These are supplied by the `--scenario-strategies` flag and tell the scenario what to test, but they are always optional. Also note you can obtain these by running `--list-scenarios`\n",
    "\n",
    "Basic usage will look something like:\n",
    "\n",
    "```shell\n",
    "pyrit_scan <scenario> --initializers <initializer1> <initializer2> --scenario-strategies <strategy1> <strategy2>\n",
    "```\n",
    "\n",
    "You can also override scenario parameters directly from the CLI:\n",
    "\n",
    "```shell\n",
    "pyrit_scan <scenario> --max-concurrency 10 --max-retries 3 --memory-labels '{\"experiment\": \"test1\", \"version\": \"v2\"}'\n",
    "```\n",
    "\n",
    "Or concretely:\n",
    "\n",
    "```shell\n",
    "!pyrit_scan foundry_scenario --initializers simple openai_objective_target --scenario-strategies base64\n",
    "```\n",
    "\n",
    "Example with a basic configuration that runs the Foundry scenario against the objective target defined in `openai_objective_target` (which just is an OpenAIChatTarget with `DEFAULT_OPENAI_FRONTEND_ENDPOINT` and `DEFAULT_OPENAI_FRONTEND_KEY`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting PyRIT...\n",
      "Loading PyRIT modules...\n",
      "Running 1 initializer(s)...\n",
      "\n",
      "Running scenario: foundry_scenario\n",
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n",
      "Error: 'charmap' codec can't encode character '\\U0001f4ca' in position 41: character maps to <undefined>\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Executing Foundry Scenario:   0%|          | 0/2 [00:00<?, ?attack/s]WARNING:pyrit.prompt_target.openai.openai_target:BadRequestError request_id=None is_content_filter=True payload={\"error\": {\"message\": \"The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering p\n",
      "ERROR:pyrit.exceptions.exception_classes:BadRequestException encountered: Status Code: 400, Message: {'error': {'message': \"The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766\", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': True, 'severity': 'high'}}}}}\n",
      "WARNING:pyrit.prompt_target.openai.openai_target:BadRequestError request_id=None is_content_filter=True payload={\"error\": {\"message\": \"The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering p\n",
      "ERROR:pyrit.exceptions.exception_classes:BadRequestException encountered: Status Code: 400, Message: {'error': {'message': \"The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766\", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': True, 'severity': 'medium'}}}}}\n",
      "\n",
      "Executing Foundry Scenario:  50%|#####     | 1/2 [00:14<00:14, 14.48s/attack]\n",
      "Executing Foundry Scenario: 100%|##########| 2/2 [00:59<00:00, 32.29s/attack]\n",
      "Executing Foundry Scenario: 100%|##########| 2/2 [00:59<00:00, 29.62s/attack]\n"
     ]
    }
   ],
   "source": [
    "!pyrit_scan foundry_scenario --initializers openai_objective_target --strategies base64"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8",
   "metadata": {},
   "source": [
    "Or with all options and multiple initializers and multiple strategies:\n",
    "\n",
    "```shell\n",
    "pyrit_scan foundry_scenario --database InMemory --initializers simple objective_target objective_list --scenario-strategies easy crescendo\n",
    "```\n",
    "\n",
    "You can also override scenario execution parameters:\n",
    "\n",
    "```shell\n",
    "# Override concurrency and retry settings\n",
    "pyrit_scan foundry_scenario --initializers simple objective_target --max-concurrency 10 --max-retries 3\n",
    "\n",
    "# Add custom memory labels for tracking (must be valid JSON)\n",
    "pyrit_scan foundry_scenario --initializers simple objective_target --memory-labels '{\"experiment\": \"test1\", \"version\": \"v2\", \"researcher\": \"alice\"}'\n",
    "```\n",
    "\n",
    "Available CLI parameter overrides:\n",
    "- `--max-concurrency <int>`: Maximum number of concurrent attack executions\n",
    "- `--max-retries <int>`: Maximum number of automatic retries if the scenario raises an exception\n",
    "- `--memory-labels <json>`: Additional labels to apply to all attack runs (must be a JSON string with string keys and values)\n",
    "\n",
    "You can also use custom initialization scripts by passing file paths. It is relative to your current working directory, but to avoid confusion, full paths are always better:\n",
    "\n",
    "```shell\n",
    "pyrit_scan encoding_scenario --initialization-scripts ./my_custom_config.py\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9",
   "metadata": {},
   "source": [
    "#### Using Custom Scenarios\n",
    "\n",
    "You can define your own scenarios in initialization scripts. The CLI will automatically discover any `Scenario` subclasses and make them available:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "# my_custom_scenarios.py\n",
    "from pyrit.common.apply_defaults import apply_defaults\n",
    "from pyrit.scenario import Scenario\n",
    "from pyrit.scenario.core.scenario_strategy import ScenarioStrategy\n",
    "\n",
    "\n",
    "class MyCustomStrategy(ScenarioStrategy):\n",
    "    \"\"\"Strategies for my custom scenario.\"\"\"\n",
    "\n",
    "    ALL = (\"all\", {\"all\"})\n",
    "    Strategy1 = (\"strategy1\", set[str]())\n",
    "    Strategy2 = (\"strategy2\", set[str]())\n",
    "\n",
    "\n",
    "@apply_defaults\n",
    "class MyCustomScenario(Scenario):\n",
    "    \"\"\"My custom scenario that does XYZ.\"\"\"\n",
    "\n",
    "    @classmethod\n",
    "    def get_strategy_class(cls):\n",
    "        return MyCustomStrategy\n",
    "\n",
    "    @classmethod\n",
    "    def get_default_strategy(cls):\n",
    "        return MyCustomStrategy.ALL\n",
    "\n",
    "    def __init__(self, *, scenario_result_id=None, **kwargs):\n",
    "        # Scenario-specific configuration only - no runtime parameters\n",
    "        super().__init__(\n",
    "            name=\"My Custom Scenario\",\n",
    "            version=1,\n",
    "            strategy_class=MyCustomStrategy,\n",
    "            scenario_result_id=scenario_result_id,\n",
    "        )\n",
    "        # ... your scenario-specific initialization code\n",
    "\n",
    "    async def _get_atomic_attacks_async(self):\n",
    "        # Build and return your atomic attacks\n",
    "        return []"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11",
   "metadata": {},
   "source": [
    "Then discover and run it:\n",
    "\n",
    "```shell\n",
    "# List to see it's available\n",
    "pyrit_scan --list-scenarios --initialization-scripts ./my_custom_scenarios.py\n",
    "\n",
    "# Run it with parameter overrides\n",
    "pyrit_scan my_custom_scenario --initialization-scripts ./my_custom_scenarios.py --max-concurrency 10\n",
    "```\n",
    "\n",
    "The scenario name is automatically converted from the class name (e.g., `MyCustomScenario` becomes `my_custom_scenario`)."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
